Computational Capabilities of Restricted Two

We study the extent to which xing the second layer weights reduces the capacity and generalization ability of a two-layer perceptron. Architectures with N inputs, K hidden units and a single output are considered, with both overlapping and non-overlapping receptive elds. We obtain from simulations one measure of the strength of a network-its critical capacity, c. Using the ansatz med / (c ?) ?2 to describe the manner in which the median learning time diverges as c is approached, we estimate c in a manner that does not depend on arbitrary impatience parameters. The CHIR learning algorithm is used in our simulations. For K = 3 and overlapping receptive elds we show that the general machine is equivalent to the Committee with the same architecture. For K = 5 and the same connectivity the general machine is the union of four distinct networks with xed second layer weights, of which the Committee is the one with the highest c. Since the capacity of the union of a nite set of machines equals that of the strongest constituent, the capacity of the general machine with K = 5 equals that of the Committee. We were not able to prove this for general K, but believe that it does hold. We investigated the internal representations used by diierent machines, and found that high correlations between the hidden units and the output reduce the capacity. Finally we studied the Boolean functions that can be realized by networks with xed second layer weights. We discovered that two diierent machines implement two completely distinct sets of Boolean functions.